InnoDB: Error: semaphore wait has lasted > 600 seconds
InnoDB: We intentionally crash the server, because it appears to be hung.
100906 11:50:29 InnoDB: Assertion failure in thread 24583 in file 
sync0arr.c line 925
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. See section 6.1 of
InnoDB: http://www.innodb.com/ibman.php about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help 
diagnose
the problem, but since we have already crashed, something is definitely 
wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=2093056
max_used_connections=14
max_connections=100
threads_connected=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections 
= 474735 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
InnoDB: Thread 712743 stopped in file ha_innodb.cc line 423
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfedf758, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80720d4
0x8250d48
0x81ed044
0x80f9148
0x824e4fc
0x828452a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://www.mysql.com/doc/en/Using_stack_trace.html and 
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.




=================


InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 409626.
InnoDB: You may have to recover from a backup.
090405 17:43:36  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex...;InnoDB: End of page dump
090405 17:43:36  InnoDB: Page checksum 357069505, prior-to-4.0.14-form checksum 850772959
InnoDB: stored checksum 3069136, prior-to-4.0.14-form stored checksum 3893751558
InnoDB: Page lsn 39 940125555, low 4 bytes of lsn at page end 2441773099
InnoDB: Page number (if stored to page already) 409626,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 462
InnoDB: (index "userid" of table "subeta_subeta"."new_forum_sub")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 409626.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.


=================

(gdb) thread apply all bt

Thread 13 (process 18993):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x10f3840, mp=0x10f2e00, file=0xbbc36b "mysqld.cc", line=3994) at thr_mutex.c:237
#2  0x00000000006a731d in main (argc=30, argv=0x7fff77fd3fc8) at mysqld.cc:3994

Thread 12 (process 18995):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x19fd318, mp=0x19fd2a0, file=0xc704aa "os/os0sync.c", line=338) at thr_mutex.c:237
#2  0x00000000009379d1 in os_event_wait (event=0x19fd2a0) at os/os0sync.c:338
#3  0x0000000000936a00 in os_aio_simulated_handle (global_segment=0, message1=0x408000f8, message2=0x408000f0, type=0x408000e8) at os/os0file.c:4144
#4  0x00000000009125b4 in fil_aio_wait (segment=0) at fil/fil0fil.c:4246
#5  0x000000000095e6bf in io_handler_thread (arg=0x1102720) at srv/srv0start.c:434
#6  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#7  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 11 (process 18996):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x19fd3f8, mp=0x19fd380, file=0xc704aa "os/os0sync.c", line=338) at thr_mutex.c:237
#2  0x00000000009379d1 in os_event_wait (event=0x19fd380) at os/os0sync.c:338
#3  0x0000000000936a00 in os_aio_simulated_handle (global_segment=1, message1=0x410010f8, message2=0x410010f0, type=0x410010e8) at os/os0file.c:4144
#4  0x00000000009125b4 in fil_aio_wait (segment=1) at fil/fil0fil.c:4246
#5  0x000000000095e6bf in io_handler_thread (arg=0x1102728) at srv/srv0start.c:434
#6  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#7  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 10 (process 18997):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x19fd4d8, mp=0x19fd460, file=0xc704aa "os/os0sync.c", line=338) at thr_mutex.c:237
#2  0x00000000009379d1 in os_event_wait (event=0x19fd460) at os/os0sync.c:338
#3  0x0000000000936a00 in os_aio_simulated_handle (global_segment=2, message1=0x418020f8, message2=0x418020f0, type=0x418020e8) at os/os0file.c:4144
#4  0x00000000009125b4 in fil_aio_wait (segment=2) at fil/fil0fil.c:4246
#5  0x000000000095e6bf in io_handler_thread (arg=0x1102730) at srv/srv0start.c:434
#6  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#7  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 9 (process 18998):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x19fd5b8, mp=0x19fd540, file=0xc704aa "os/os0sync.c", line=338) at thr_mutex.c:237
#2  0x00000000009379d1 in os_event_wait (event=0x19fd540) at os/os0sync.c:338
#3  0x0000000000936a00 in os_aio_simulated_handle (global_segment=3, message1=0x420030f8, message2=0x420030f0, type=0x420030e8) at os/os0file.c:4144
#4  0x00000000009125b4 in fil_aio_wait (segment=3) at fil/fil0fil.c:4246
#5  0x000000000095e6bf in io_handler_thread (arg=0x1102738) at srv/srv0start.c:434
#6  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#7  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 8 (process 19000):
#0  0x00002b0b33ac2fc2 in select () from /lib/libc.so.6
#1  0x00000000009380da in os_thread_sleep (tm=1000000) at os/os0thread.c:281
#2  0x000000000095d156 in srv_lock_timeout_and_monitor_thread (arg=0x0) at srv/srv0srv.c:1905
#3  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#4  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 7 (process 19001):
#0  0x00002b0b33ac2fc2 in select () from /lib/libc.so.6
#1  0x00000000009380da in os_thread_sleep (tm=2000000) at os/os0thread.c:281
#2  0x000000000095d6c9 in srv_error_monitor_thread (arg=0x0) at srv/srv0srv.c:2131
#3  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#4  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 6 (process 19002):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x1373738, mp=0x13736c0, file=0xc704aa "os/os0sync.c", line=338) at thr_mutex.c:237
#2  0x00000000009379d1 in os_event_wait (event=0x13736c0) at os/os0sync.c:338
#3  0x000000000095e018 in srv_master_thread (arg=0x0) at srv/srv0srv.c:2620
#4  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#5  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 5 (process 19003):
#0  0x00002b0b32cfee29 in do_sigwait () from /lib/libpthread.so.0
#1  0x00002b0b32cfeecd in sigwait () from /lib/libpthread.so.0
#2  0x00000000006a25cb in signal_hand (arg=0x0) at mysqld.cc:2490
#3  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#4  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 4 (process 19038):
#0  0x00002b0b32cfb7a6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x0000000000acd474 in safe_cond_wait (cond=0x1b7b858, mp=0x1b7b7e0, file=0xc4c275 "event_scheduler.cc", line=759) at thr_mutex.c:237
#2  0x000000000086cd7c in Event_scheduler::cond_wait (this=0x1b7b7e0, thd=0x1b595d8, abstime=0x0, msg=0xc4c55b "Waiting scheduler to stop", func=0xc4cb09 "stop", line=652) at event_scheduler.cc:759
#3  0x000000000086d2c9 in Event_scheduler::stop (this=0x1b7b7e0) at event_scheduler.cc:652
#4  0x0000000000877f6c in Events::switch_event_scheduler_state (new_state=Events::EVENTS_OFF) at events.cc:1087
#5  0x00000000006c4262 in sys_var_event_scheduler::update (this=0x10f4920, thd=0x1b595d8, var=0x1b910c8) at set_var.cc:3767
#6  0x00000000006c0492 in set_var::update (this=0x1b910c8, thd=0x1b595d8) at set_var.cc:3177
#7  0x00000000006c5f7f in sql_set_variables (thd=0x1b595d8, var_list=0x1b5b588) at set_var.cc:3062
#8  0x00000000006b7858 in mysql_execute_command (thd=0x1b595d8) at sql_parse.cc:3130
#9  0x00000000006bc557 in mysql_parse (thd=0x1b595d8, inBuf=0x1b90f18 "SET GLOBAL event_scheduler= OFF", length=31, found_semicolon=0x44088f10) at sql_parse.cc:5533
#10 0x00000000006bd2f9 in dispatch_command (command=COM_QUERY, thd=0x1b595d8, packet=0x1b81199 "SET GLOBAL event_scheduler= OFF", packet_length=31) at sql_parse.cc:1027
#11 0x00000000006be620 in do_command (thd=0x1b595d8) at sql_parse.cc:765
#12 0x00000000006ac63e in handle_one_connection (arg=0x1b595d8) at sql_connect.cc:1133
#13 0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#14 0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 3 (process 29120):
#0  code_state () at dbug.c:329
#1  0x0000000000ad10d2 in _db_enter_ (_func_=0xcacf4f "_myfree", _file_=0xcace00 "safemalloc.c", _line_=260, _sfunc_=0x4414c010, _sfile_=0x4414c008, _slevel_=0x4414c004, _sframep_=0x4414bff8) at dbug.c:942
#2  0x0000000000aad43f in _myfree (ptr=0x1b89128, filename=0xbb879f "sql_class.cc", lineno=688, myflags=64) at safemalloc.c:260
#3  0x000000000068d398 in THD::cleanup (this=0x1ba33e8) at sql_class.cc:688
#4  0x00000000006a3636 in unlink_thd (thd=0x1ba33e8) at mysqld.cc:1767
#5  0x00000000006a36dc in one_thread_per_connection_end (thd=0x1ba33e8, put_in_cache=true) at mysqld.cc:1849
#6  0x00000000006ac6ae in handle_one_connection (arg=0x1ba33e8) at sql_connect.cc:1140
#7  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#8  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10

Thread 2 (process 29121):
#0  0x00002b0b32cfb5dd in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0x000000000086d445 in ~Event_scheduler (this=0x1b7b7e0) at event_scheduler.cc:360
#2  0x0000000000878189 in Events::deinit () at events.cc:975
#3  0x00000000006a79a5 in close_connections () at mysqld.cc:871
#4  0x00000000006a7e01 in kill_server (sig_ptr=0x0) at mysqld.cc:1065
#5  0x00000000006a7e41 in kill_server_thread (arg=0x44048138) at mysqld.cc:1094
#6  0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#7  0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10
Current language:  auto; currently c

Thread 1 (process 19361):
#0  0x00002b0b32cfc422 in pthread_kill () from /lib/libpthread.so.0
#1  0x000000000083f2fd in write_core (sig=6) at stacktrace.c:240
#2  0x00000000006a3408 in handle_segfault (sig=6) at mysqld.cc:2293
#3  <signal handler called>
#4  0x00002b0b33a276a5 in raise () from /lib/libc.so.6
#5  0x00002b0b33a29100 in abort () from /lib/libc.so.6
#6  0x0000000000acd023 in safe_mutex_lock (mp=0x1b7b7e0, try_lock=0 '\0', file=0xc4c275 "event_scheduler.cc", line=702) at thr_mutex.c:103
#7  0x000000000086cc3d in Event_scheduler::lock_data (this=0x1b7b7e0, func=0xc4cb0e "is_running", line=587) at event_scheduler.cc:702
#8  0x000000000086ce2f in Event_scheduler::is_running (this=0x1b7b7e0) at event_scheduler.cc:587
#9  0x000000000086db97 in Event_scheduler::run (this=0x1b7b7e0, thd=0x1ba7188) at event_scheduler.cc:469
#10 0x000000000086e076 in event_scheduler_thread (arg=0x1bb39a8) at event_scheduler.cc:235
#11 0x00002b0b32cf7317 in start_thread () from /lib/libpthread.so.0
#12 0x00002b0b33ac9c7d in clone () from /lib/libc.so.6
Cannot access memory at address 0x2b0b33ac9c10
(gdb) 



====


291 pthread_cond_wait@@GLIBC_2.3.2,one_thread_per_connection_end,handle_one_connection
 57 read,my_real_read,my_net_read,do_command,handle_one_connection,start_thread
 26 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,os_aio_simulated_handle,fil_aio_wait,io_handler_thread,start_thread
  3 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_purge_worker_thread
  1 select,os_thread_sleep,srv_purge_thread
  1 select,os_thread_sleep,srv_master_thread
  1 select,os_thread_sleep,srv_lock_timeout_and_monitor_thread
  1 select,os_thread_sleep,srv_error_monitor_thread
  1 select,handle_connections_sockets,main,select
  1 read,vio_read_buff,my_real_read,my_net_read,cli_safe_read,handle_slave_io
  1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,sync_array_wait_event,rw_lock_s_lock_spin,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_index_entry,row_upd_step,row_update_for_mysql,ha_innodb::delete_row,handler::ha_delete_row,mysql_delete,mysql_execute_command,mysql_parse,Query_log_event::do_apply_event,apply_event_and_update_pos,handle_slave_sql
  1 pread64,os_file_pread,os_file_read,fil_io,buf_read_page_low,buf_read_page,buf_page_get_gen,btr_cur_search_to_nth_level,row_search_for_mysql,ha_innodb::index_read,handler::index_read_idx_map,join_read_const,join_read_const_table,make_join_statistics,JOIN::optimize,mysql_select,handle_select,execute_sqlcom_select,mysql_execute_command,mysql_parse,dispatch_command,do_command,handle_one_connection
  1 do_sigwait,sigwait,signal_hand


